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ABSTRACT 

A research plan for investigating the effects of 
student portfolios on teaching and learning at three different school 
sites is described. Th^ plan accommodates two categories of research 
questions: (1) broad, cross-site questions; and (2) site-specific 
questions generated by participants at each individual site^ Data 
collection strategies includes (1) individual and fvocus group 
interviews with students, teachers, site administrators, and parents; 
(2) questionnaires; (3) classroom observations; and (4) analysis of 
the student portfolios. Data analysis will be ongoing and shared with 
participants in a cycle of collaboration that is intended to inform 
both the research process and portfolio implementation efforts at the 
sites. This information is intended to present researchers and 
practitioners with several models for using student portfolios, and 
also to provide information on the ways that student portfolios 
affect teaching and learning. The results of this year-long 
investigation will be reported in August of 1992 and yearly 
thereafter as the study is extended. (SLD) 
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Research Design for Investigating 
the Effects of Student Portfolios on Teaching and Learning 



This document describes a plan for conducting field-based research 
on the use of student portfolios for assessing student performance. 
Three different sites, in which portfolios are actively in use, will be 
studied for one school year to determine the ways in which the use 
of student portfolios affects learning and teaching. The information 
gained at each of these individual sites will be used to identify cross- 
site themes and broader issues in the use of student portfolios. 

The first section of the document discusses the research questions 
that will be investigated in this study. The next sections present the 
data collection and analysis strategies that will be used to address 
these questions. The following section presents the plan for 
documenting the portfolio processes and products. In the final 
section, criteria for site selection is discussed. 

Research Questions 

The design challenge is to develop a plan that flexibly accommodates 
the variety in specific site activities and contexts, and, at the same 
time, addresses critical issues in the use of student portfolios that cut 
across individual sites and particular portfolio approaches. The 
design must be responsive both to the needs of the individual pilot 
sites and to the overall research program. 
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With that challenge in mind, the following section describes a two- 
pronged research strategy in which a set of cross-site research 
questions that guides data collection across all of the individual sites 
is complemented by a set of research questions that is site-specific 
and generated by the individual sites. Collecting data on the same 
set of key research questions across each of the sites will allow the 
research to identify broadly the ways that alternative forms of 
student assessment affect teaching and learning, independent of 
particular sites or instruments, while collecting data on questions 
that emerge at the individual pilot sites will help to make the 
research more responsive to the needs of each pilot site, and will 
ultimately strengthen the individual assessment instruments. 

Cross-site research questions include (but are not limited to): 

• Effects on student learning, especially for at-risk students 

• Effects on instructional practices 

• Effects on professional development of teachers 

• Costs and benefits of assessment 

• Effects on school-level restructuring and reform 

• Effects on large-scale assessment and accountability 

Site-specific and site-gencated questions could include questions 
such as: 

• Effects of student portfolios on math versus science classes 

• Effects of portfolios on special populations (e.g., ESL) 

• Effects of process writing portfolios on science instruction 

• Effects of portfolios on student self-evaluation strategies 



• Effects of portfolios on student attitudes 

• Effects of portfolio reporting procedures on parents 

The development of the both the cross-site and the site-specific 
questions will be an iterative and recursive process in which the 
research questions are continually reframed and refined in light of 
the data that are collected. 

The cross-site questions, broad and general in focus, are intended as 
frames for a more specific set of related questions. The research 
question on the "effects of student learning," for example, frames a 
set of questions that deals with issues such as the effects of portfolios 
on student attitudes, on student self-assessment, on student writing 
skills, on student motivation. The specific questions that emerge 
under each of these cross-site questions will be determined in part 
by the different emphases at each portfolio site. Student 
collaboration may be a goal at one site, and, consequently, an area 
that will receive a great deal of attention. At another site, student 
self-assessment may be a key area of interest. 

The site-generated questions obviously cannot be determined ahead 
of time. These research questions will be shaped by the individual 
needs and interests of the participants and the particular context in 
which the portfolios are used. Participants may have clearly formed 
research questions at the outset of this research, or these questions 
may emerge as the process of using portfolios unfolds. The role of 
the researcher in this situation is to stimulate and support the 
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participants in framing and investigating tnanaf^eable and 
meaningful research questions. 

Question Levek 

Both the cross-site and site-based research questions need to be 
addressed at two different levels: 

• The instrument 

• The institution 

Gathering information about the effectiveness of the individual pilot 
project instruments is the main goal of this investigation. At the 
same time, it is also imperative to gather information about the ways 
that the institutiona l context facilitates or hinders the 
implementation of the assessment instrument. A portfolio project 
that succeeds at one school, for example, may be a dismal failure at 
another. Only by collecting information about the ways that the 
institutional context affects the implementation of the assessment 
instrument can a determination be made of the conditions and 
support that are necessary for successful implementation of 
alternative forms of student assessment. 

Data Collection Strategies 

To address these research questions, it is necessary to collect a 
variety of information about the various portfolio projects and 
institutional ontexts. The following data collection strategies are 
proposed: Initial survey, interviews, c uestionnaires. observations, 
and reviews of student portfolios. 



Data Collection Strategies 

• Initial survey 

Site characteristics 
Instrument characteristics 

• Interviews 

Administrators 
Teachers (focus groups) 
Students (focus groups) 

• Questionnaires 

Teachers 
Students 
Parents 

• Observations 

Classroom implementation 
Portfolio-related staff sessions 

• Portfolio reviews 

Review of Individual portfolio with student 
Review of class portfolios with teacher 
Analysis of portfolio contents 

Initial Survey 

The initial survey will be conducted as early as possible in the school 
year. The purpose of the initial survey is to gather information 
about the site characteristics and the assessment instrument. The 
site characteristics survey will focus on identifying the participants 
and context of the assessment, documenting past uses of the 
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portfolio, and learning about future plans for developing and 
implementing the portfolio. The instrument characteristics survey 
will aim at identifying portfolio procedures and collecting portfolio 
materials. 

Interviews 

Interviews will be held with individual participants and with focus 
groups. The school administrator and project director will be 
individually interviewed. District administrators may be 
individually interviewed as well, if appropriate. Teachers and 
students will be interviewed in focus groups. Focus groups of three 
to five participants will be a more economical use of teachers and 
students time than individual interviews with each student and 
teacher. In addition, the dynamic interaction among the participants 
is likely to produce more insightful responses than individual 
interviews. Interviews will be conducted at least three times during 
the school year. 

The same et of cross-site q uestions will be investigated at each of 
the three sites; however, the participants* responses — and the 
subsequent follow-up questions and prooes — to these broad, initial 
questions may take distinctly different directions at each of the three 
sites. Responses to the question on the effects on student learning, 
for example, may focus on student attitudes at one site, while at 
another site the focus may be on students* ability to evaluate their 
own progress. To allow for the investigation to address the most 
significant issues from the participants* perspective and to keep open 
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■ the possibility to explore unanticipated issues, a "funnel approach*' 
m will be taken with the interviews. In this approach, respondents are 
• first asked broad questions, with subsequent questions gradually 

■ narrowing in focus in light of participants' responses. 

P Questionnaires 

^ Brief questionnaires will be administered to a representative sample 

■ of participants at the site. The questionnaires will ask participants to 

■ rate (on a four point scale) and comment on the effects of the 
portfolios on the instructional program. For students, the 

I questionnaire will focus on the effects of the portfolio on their 

— learning. Students will be asked to rate the effects of the portfolio on 

■ various aspects of their learning, such as content learning, attitudes, 

■ and self-assessment. For teachers, the questionnaire will focus on 
the effects of the portfolios on instructional practices. Similar 

I questionnaire forms that combine ratings and comments will be used 

for administrators and parents. 

■ Observations 

Two kinds of observations will be conducted: observations of 
I classrooms and portfolio-related staff sessions. The classroom 

observations will examine the ways that the portfolio is actually used 

■ by teachers and students on a day-to-day basis. These observations 

■ will be scheduled with the classroom teacher around both everyday 

portfolio activities and special events, such as formal portfolio 

p reviews with students or portfolio celebrations. Observations will r 

I 
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■ also be conducted of significant portfolio-related staff sessions, such 
M as portfolio development meetings or portfolio evaluation sessions. 

■ Portfolio Reviews 

Portfolio review sessions will be held with a sample of students and 
I teachers. In addition, the contents of a representative sample of 

^ student portfolios will be analyzed. Portfolio review sessions with 

■ students will consist of the researcher and an individual student 

■ examining and discussing the contents of the student's portfolio. 
Students will be asked what their work reveals about their learning. 

P Portfolio review sessions with teachers will consist of the researcher 

^ and teacher examining and discussing the contents of several 

■ different student portfolios. Teachers will be asked about the ways 

■ that the portfolio affects their instruction for that student. 

I Research Design Matrix 

The following matrix links the project research questions with data 

■ collection strategies and can be used to guide the data collection 

■ process and development of individual data collection instrumepts. 
For example, for obtaining information necessary to answer the 

H question on "effects of student portfolio assessment on teaching," the 

primary sources of information will be the interviews with classroom 
m teachers. A good secondary source is the students. For answering 

■ the question about "the effects of portfolio assessment on 
accountability," for example, the primary source of information will 

■ be the school principal or district administrator'^ Teachers are a 
good secondary source of information for answering this question. 
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MATRIX: RESEARCH QUESTION BY DATA COLLECTION STRATEGY 
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Key 

X = primary source of infonnation about research question 
✓ - secondary source of information about question 
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Data Analysis 

Data analysis will rely on both quantitative and qualitative analyses. 
Quantitative data analysis procedures will be used to analyze student 
outcome data that are reported in numerical form. Qualitative data 
analysis procedures will be used to analyze all forme of data 
collected, including student portfolios, survey information, 
interviews, questionnaires, and observations. 

Data will be analyzed on an ongoing basis for emerging themes. 
These emerging themes will be identified and then discussed with 
project participants. This cycle of collaboration between researcher 
and participants, in which participants comment on the emerging 
themes that have been identified by the researcher, will not only 
enrich and inform the research ftndings, , it will also strengthen the 
portfolio process itself by providing the participants with ongoing 
information about the effectiveness of student portfolios. This cycle 
of identifying and reviewing themes with the participants will be an 
iterative process that will continue throughout the study. 

The portfolio process and products will be examined for issues of 
reliability, validity, and potential sources of bias. Reliability studies 
will include measures of inter-rater reliability (if appropriate) and 
reliability of student outcomes. Validity studies will include 
reviewing the portfolio methods and materials for congruence with 
curriculum goals. Potential sources of bias will be studied through 
interviews with students and teachers, and by examining the results 
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of the assessments for differences in scores or reported performance 
among students by categories such as race, ethnicity, and language. 

Instrument Documentation 

This research and development effort has two goals: (1) to explore 
the effectiveness of portfolio approaches to assessing student 
achievement and (2) to support and document the work of the pilot 
projects in developing their student portfolios. The previous section 
described the research plan for investigating the effectiveness of 
portfolio assessment at each site as well as across the group of sites; 
the following section describes the research plan for documenting the 
portfolio processes and products at each site. 

An important task of this research is to document both the processes 
of developing and implementing each individual portfolio approach 
as well as the portfolio products that are created at each site. 
Documentation of the processes will allow others to learn from the 
strategies that each site used in developing and administering their 
assessment instrument, while documentation of the products will 
enable the assessment instruments to be studied and potentially 
disseminated for use at other sites. 

Documentation includes process notes and product materials on the 
following features of the assessment instrument: 

• Goals of Assessment 

• Performance Standards 
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• Portfolio Approach 

• Scoring Criteria 

• Scoring Procedures 

• Reporting Methods 

• Training Strategies (if appropriate) 

Information on the processes will have a different focus from the 
information about the products. Questions about the process ask how 
and why; questions about the product ask what . Questions about 
process ask: How and why were these goals chosen? How was the 
portfolio developed? Questions about product ask: What are the 
goals of the assessment? What are the assessment tasks? Collecting 
information about both process and product will enable others to 
examine the assessment instrument itself as well as consider 
important issues that arose in the development and administration of 
the instrument. 

Site Selection 

We will select three different sites in which student portfolios are in 
use. While these three sites will all share a similar focus on 
portfolios, the selection process will aim to maximize the differences 
across sites in terms of grade level, curricular focus, and portfolio 
approach. Selecting three sites that differ from each other along 
these dimensions will allow for a broader view of portfolios and their 
effects on learning and teaching. 

The following criteria will be observed in selecting the sites: 
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(1) Significant proportion of "at-risk" st udents at school site. 

Ii roving the education of "at-risk" students is a central aim of this 
research. A serious criticism of traditional forms of student 
assessment, such as standardized tests, is that these assessments are 
culturally biased and that they do not recognize the diversity in 
ways of communicating and thinking. Portfolio- based assessment 
has been touted as one method for overcoming these biases because 
portfolios allow for multiple forms of expression. This assertion, 
while compelling and attractive, is as yet unproven. The use of 
portfolios may "open up" assessment and reduce the possibility of 
cultural bias; on the other hand, portfolios may be subject to some of 
the same problems as traditional tests, or, given the subjective 
nature of portfolio assessment, even graver problems. For these 
reasons, research on the effects of portfolio assessment on the 
learning of "at-risk" students; is urgently needed. 

(2) Significant nronortion of teachers in the school involved in the 
use of student portfolios Studying the use of portfolios by individual 
teachers is important; studying the use of student portfolios by an 
entire school staff, however, can provide insights into the effects of 
portfolios on learning and teaching that can not be gained by 
studying individual efforts. When a school staff embraces student 
portfolios as an approach to assessment, this decision can have 
profound effects on such areas as the curriculum goals of the school, 
the professional development of the teachers, and the participation 
of parents in school activities. The t^r. of portfolios by a school staff 
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not only requires a school-wide conversation about learning and 
teachings it also provides teachers with an occasion and a structure 
for discussing pedagogically substantive matters. 



(3) Commitment to stu dent portfolios by the school principal . 
Without the commitment of the school principal, it is unlikely that a 
school-wide portfolio effort will sustain the necessary momentum for 
success. Teachers and students need the time and support to 
develop and implement portfolios in their classrooms. If portfolios 
are an added-on activity, teachers and students will likely not be 
able to find the necessary time to maintain them. If portfolios are 
integrated and embedded in the school's procedures so that they 
replace some of what teacher routinely do, they stand a much better 
chance of succeeding. If portfolios, for example, replace the 
traditional report card, teachers will be better able to find the time 

to effectively use them, students will be more motivated to construct 
meaningful portfolios, and parents will be more interested in what 
the portfolio reveals about their children's knowledge and skills. 
Principals are crucial in making the portfolio an integral part of the 
school program. 

(4) Demonstrated prior commitment to and experience with student 
portfolios. Only sites that have already initiated portfolio assessment 
procedures for at least one year will be considered for inclusion in 
this study. This criterion is both a measure of school-wide 
commitment to and experience with student portfolios. Both 
motivation and previous involvement with portfolios are necessary 
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I for this research on the effects of portfolios to take place. The first 

year of developing and implementing portfolios is often an 
• exploratory effort that usually does not provide coherent information 

■ about the effects of portfolios on learning and teaching. Developing a 
school-wide student portfolio assessment system at the same time 

I that it is being implemented is akin to building an airplane while 

trying to fly it. These early, exploratory efforts are less likely to 
I provide coherent information about the effects of portfolios on 

■ teaching and learning. School staff with two or three years 
experience with student portfolios, however, usually have worked 

I out most of the portfolio design problems and are ready to 

systematically implement portfolios school-wide and thoughtfully 
I considered their consequences on learning and teaching. 

" Conclusion 

■ This document describes a research plan for investigating the effects 
of student portfolios on teaching and learning at three different 

I school sites. The plan provides for two categories of research 

^ questions: broad, cross-site questions and site-specific questions 

■ generated by participants at each individual site. Data collection 

■ strategies include individual and focus group interviews with 
students, teachers, site administrators, and (possibly) parents; 

I questionnaires; classroom observations; and analyses of the student 

g portfolios. Data analysis will be ongoing and shared with participants 

■ in a cycle of collaboration that is intended to inform both the 

■ research process and portfolio implementation efforts at the sites. 
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This information is intended to present researchers and practitioners 
with several models for using student portfolios as an assessment 
approach, and also to provide information on the ways that student 
portfolios affect teaching and learning. The results of this year-long 
investigation will be reported in August of 1992 (and yearly 
thereafter if the study extends into future years). 
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